3D face authentication and recognition based on bilateral symmetry analysis

ABSTRACT

There is provided a novel approach for automatic human face authentication. Taking a 3D triangular facial mesh as input, the approach first automatically extracts the bilateral symmetry plane of the face surface. The intersection between the symmetry plane and the facial surface, namely the Symmetry Profile, is then computed. By using both the mean curvature plot of the facial surface and the curvature plot of the symmetry profile curve, three essential points of the nose on the symmetry profile are automatically extracted. The three essential points uniquely determine a Face Intrinsic Coordinate System (FICS). Different faces are aligned based on the FICS. The Symmetry Profile, together with two transversal profiles, namely the Forehead Profile and the Cheek Profile compose a compact representation, called the SFC representation, of a 3D face surface. The face authentication and recognition steps are finally performed by comparing the SFC representation of the faces.

RELATED APPLICATION DATA

This application is based on and claims the benefit of U.S. ProvisionalPatent Application No. 60/577,367 filed on Jun. 3, 2004, the disclosureof which is incorporated herein by this reference.

U.S. GOVERNMENT FINANCIAL ASSISTANCE

Financial assistance for this project was provided by the United StatesGovernment, National Science Foundation Grant No. 0312849. The UnitedStates Government may own certain rights to this invention.

BACKGROUND

This invention relates to automatic face authentication and recognition.More particularly, it relates to a method and system for authenticationand recognition of faces using a three-dimensional facial surfacerepresentation and facial bilateral symmetry plane extraction to derivea profile curve and a coordinate system for aligning facialrepresentations for comparison.

Automatic face authentication refers to using facial images or scans toverify an identity claim of a known individual. Automatic faceauthentication has long been an active research area for its widepotential applications, such as law enforcement, security access, andman-machine interaction. Authentication involves performing verificationbased on a one-to-one search to validate the identity claim of anindividual (i.e., access control for a building, room, or for making atransaction at an ATM terminal). Automatic face recognition refers tousing facial images or scans to identify an unknown individual within adatabase of known individuals. Recognition in one-to-many searches isbased on comparison to a database of known individuals (e.g., lawenforcement, surveillance, and recently driver licenses). Faceauthentication is in one sense a simpler process than face recognition:comparisons are made only to the claimed identity, and a threshold ofsimilarity is used to accept or reject the claim. In another sense,authentication is more difficult, because of the need to determine thisthreshold rather than using a “best match” criterion as in many facerecognition applications. With face authentication, the group of invalidIDs (imposters) is, by definition, not in the reference database.Therefore, face authentication methods must successfully operate in1-to-1 comparisons, without knowledge of possible errors in claims(i.e., who else might the individual be).

Several approaches have been promoted to recognize and authenticate anindividual or a group of people. Access control applicationsauthenticate by physical appearance (by guard personnel, receptionist);by something the individual knows (pins, passwords); by something theindividual has (lock/key, card, badge, token); by biometric evidence (aunique physiological or behavioral characteristic of the individual); orby a combination of both “what one has” (i.e., a card) and “what oneknows” (i.e., their passcode). Most workplace entry points are typicallycontrolled by a badge/card or by physical appearance. All of thesemethods, except biometrics, are fallible and can be circumvented, lost,or stolen. Interest in authentication using biometrics is thereforegrowing dramatically.

Biometric access control uses measurable physiological or behavioraltraits to automatically authenticate a person's identity. Biometriccharacteristics must be distinctive of an individual, easily acquiredand measured, and comparable for purposes of security validation. Thecharacteristic should change little over time (i.e., with age orvoluntary change in appearance) and be difficult to change, circumvent,manipulate, or reproduce by other means. Typically, high-level computerbased algorithms and database systems analyze the acquired biometricfeatures and compare them to features known or enrolled in the database.The mainstream biometric technologies use morphological featurerecognition such as fingerprints, hand geometry, iris and retinascanning, and two dimensional (2D) face authentication. Each of theseexcept face authentication is either intrusive or fails in some cases(e.g., about 10% of population do not have good enough fingerprints).

There has been a large body of literature on 2D face recognition andauthentication. For an overview, see R. Chellappa, C. Wilson, and S.Sirohey. Human and machine recognition of faces: A survey, Proceedingsof the IEEE, 83(5):705-740 (1995). Among various approaches, PrincipalComponents Analysis (PCA) to face imaging, popularly called eigenfaces,is now a cornerstone in face recognition. For a more detailedexplanation of PCA, see Turk, M., Pentland, A, Face recognition usingeigenfaces, Proc. CVPR, 1991, pp 586-591. 2D face authentication, thoughless intrusive than other biometric technologies, has simply notattained the degree of accuracy necessary in a security setting. 2D facerecognition methods are in general unable to overcome the problemsresulting from illumination, expression or pose variations, facial hairand orientation.

The emerging trend noted by many researchers in the field of facerecognition is the 3D technology, which offers several additionaladvantages to 2D face recognition. 3D technology is expected to be moreaccurate and able to overcome the problems of 2D methods, because 3Dinformation is viewpoint and lighting condition independent. There areseveral strategies in 3D face recognition. Some researchers try tosegment the 3D face surface into meaningful physiological points, linesand regions based on curvature analysis at each point. For example,Hallinan et al. utilized curvature properties to segment a face surfaceinto regions, and a set of twelve features were extracted for facerecognition. P. Hallinan, G. G. Gorden, A. L. Yuille, et al., Two andthree-dimensional patterns of the face, A. K. Peters (ed), A K PetersLtd (1999). Moreno et al., used a HK segmentation (based on the analysisof signs of mean and Gaussian curvatures at each point) to isolateregions of pronounced curvature, and up to eighty-six descriptors wereobtained from the segmented regions. A. B. Moreno, Á. Sánchez, J. F.Vélez, et al., Face recognition using 3D surface-extracted descriptors,Proceedings of the 7^(th) Irish Machine Vision & Image ProcessingConference, Antrim, N. Ireland, 2003. For the presence of noise,expression variance and incomplete scanning, however, curvature basedfeature extraction is not robust enough for face recognition. Forexample, Moreno et al. reported only a 78% correct recognition rate.

In some methods, 3D face modeling has been used as an enhancement of 2Danalysis methods. For example, Blanz et al used a 3D morphable facemodel, which is learned from a set of textured 3D scans of heads, toencode images. V. Blanz, T. Vetter. Face Recognition Based on Fitting a3D Morphable Model, IEEE Transactions on Pattern Analysis and MachineIntelligence, 25(9) (2003). Recognition is performed based on the modelcoefficients created in the process of fitting the morphable model toimages. Lee et al. also presented a model-base face recognition approachunder a similar framework. M. W. Lee, S. Ranganath, Pose-invariant facerecognition using a 3D deformable model, Pattern Recognition,36:1835-1846 (2003). In their method, the deformable 3D face model is acomposite of an edge model, a color region model and a wireframe model.This strategy is a 2D solution in nature, for the media to be comparedin these studies is still 2D intensity images. A problem with thisstrategy is that fitting the morphable model to images is acomputational expensive process. As reported by Lee et al., 4.5 minutesare needed for fitting the model to an image on a workstation with a 2GHz Pentium4 processor.

Chang et al. used both 2D and 3D face information for the recognitiontask. K. I. Chang, K. W. Bowyer, P. J. Flynn, Face Recognition Using 2Dand 3D Facial Data, The Proceedings of Workshop in Multimodal UserAuthentication, pp. 25-32, Santa Barbara, Calif., USA (2003). In theirexperiments, a PCA-based approach was tuned for face recognition from 2Dintensity images and 3D range images, respectively. Their comparisonresult is that 3D outperforms 2D. By combining the 2D distance metricand the 3D distance metric, a 2D-plus-3D criterion is used during thedecision process of face recognition. In their experiments, posevariations that occur during the acquisition process are manuallynormalized. The recognition rate of the combination scheme was reportedto be higher than 98% under the condition that the 2D and 3D images aretaken in a front view and the subjects are imaged in a normal facialexpression. The scheme of Chang et al., however, requires manualnormalization and has to use normal facial expressions.

The work of Bronstein et al, focused on developing a representation ofthe facial surface, invariant to different expressions and postures ofthe face. A. Bronstein, M. Bronstein, and R. Kimmel, Expressioninvariant 3D face recognition, Audio and Video Based Biometric PersonAuthetication, pp. 62-69 (2003). Before using the basic idea of PCA,they calculate the geometric invariants of a face surface by usingmultidimensional scaling (MDS). For a discussion of MDS, see Schwartz,E. L., Shaw, A., Wolfson, E., A numerical solution to the generalizedmapmaker's problem: flattening nonconvex polyhedral surfaces, IEEETrans. PAMI, 11: 1005-1008 (1989). Bronstein et al. did not report therecognition rate, though they claimed that their algorithm can recognizethe difference of twins. Although they did not discuss in detail thecomputation cost of their method, it appears to be high because MDSneeds to calculate the geodesic distances between each pair of points onthe surface, as well as the eigen decomposition of a large matrix.

Chua et al. analyzed over four expressions of each person to determinethe rigid parts of the face. C., F. Han, Y. Ho, 3D Human FaceRecognition Using Point Signature, 4^(th) IEEE International Conferenceon Automatic Face and Gesture Recognition, Grenoble, France, (2000).These rigid parts are modeled by point signatures for face indexing.Their method, however, was tested on only six individuals.

Taking a different approach, Beumier et al. developed an integrated 3Dface acquisition and comparison system. C. Beumier, M. Acheroy,Automatic 3D face authentication. Image and Vision Computing, 18:315-321(2000). The structured light was used to capture the facial surface. Forfacial surface comparison, they abandoned feature extraction butcalculated the global matching error of the facial surfaces. AnIterative Condition Mode (ICM) optimization was performed to determinethe rotation and translation transform that minimizes the globalmatching error sampled at fifteen profiles. In order to speed up theglobal matching process, they further extracted the central profile withmaximal protrusion (due to the nose). The central profile and a meanlateral profile were used to compare two faces in the curvature space.The main advantages of this method are its high speed and low storageneeds. But as the authors pointed out, the optimization procedure usedfor the 3D face comparison can fail due to noise, local minima or badinitial parameters. The reported Equal Error Rate, i.e, the rate atwhich false acceptances (i.e., incorrectly accepting an imposter claim)and false rejects (i.e., incorrectly rejecting a valid claim) are equal(the two rates tend to be inversely rated), is 9%. So they resorted tomanual refinement for surface matching. Cartoux et al. presented asimilar approach to extract the symmetry profile by looking for thebilateral symmetry axis of Gaussian curvature values of the facialsurface. J. Y. Cartoux, J. T. Lapreste, M. Richetin, Faceauthentification or recognition by profile extraction from range images,IEEE Computer Society Workshop on Interpretation of 3D Scenes, pp194-199 (1989).

There is a need, therefore, for an improved method and system forauthentication and recognition using 3D facial data that iscomputationally faster and more accurate in matching facial features.

Additional objects and advantages of the invention will be set forth inthe description that follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and obtained by means ofthe instrumentalities and combinations pointed out in the appendedclaims.

SUMMARY

To achieve the foregoing objects, and in accordance with the purposes ofthe invention as embodied and broadly described in this document, thereis provided a novel system and method for automatic face authentication.The system and method utilize a 3D triangular facial mesh surface as aninput, and automatically extract a bilateral symmetry plane of the facesurface. The intersection between the symmetry plane and the facialsurface, namely the Symmetry Profile, is then computed. By using boththe mean curvature plot of the facial surface and the curvature plot ofthe symmetry profile curve, three essential points of the nose on thesymmetry profile are automatically extracted. The three essential pointsuniquely determine a Face Intrinsic Coordinate System (FICS). Differentfaces are aligned based on the FICS. The Symmetry Profile, together withtwo transversal profiles, namely the Forehead Profile and the CheekProfile, comprise a compact representation, called the SFCrepresentation, of a 3D face surface. The face authentication andrecognition steps are performed by comparing the SFC representation ofthe faces. The system and method of our invention provide greatlyenhances accuracy in comparison to 2D methods and overcomes the barrierof computational complexity in comparing 3D facial data.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate the presently preferredembodiments of the invention and, together with the general descriptiongiven above and the detailed description of the preferred methods andembodiments given below, serve to explain the principles of theinvention.

FIG. 1 is a functional block diagram of an exemplary computer system forarchiving, cataloging, query, comparison and retrieval of informationrelating to 3D facial features in accordance with the present invention.

FIG. 2 shows an exemplary XML schema according to the present inventionto support the archiving, cataloging, query, comparison and retrieval of3D facial features according to the present invention.

FIG. 3 shows an example of a 3D camera scan (FIG. 3A) and itscorresponding triangle mesh geometry (FIG. 3B).

FIG. 4 illustrates the enrollment process for archiving 3D facial datain the database of the system of FIG. 1.

FIG. 5 illustrates the general operation of the system of FIG. 1 toauthenticate a facial image or scan of an individual by comparing itwith archived 3D data to verify the identity claim of the individual.

FIG. 6 is a diagram of the steps of a preferred method forauthenticating a face according to the present invention using athree-dimensional facial surface representation and facial bilateralsymmetry plane extraction to derive a profile curve and a coordinatesystem for aligning facial representations for comparison.

FIG. 7 shows results of the estimated mirror plane (FIG. 7A) and theextracted symmetry plane (FIG. 7B) for four facial meshes.

FIG. 8A-8H illustrate the MarkSkirt process used for correct for errorsin calculation of the symmetry plan than can be caused by irregularboundaries of a facial mesh. FIG. 8A shows a 3D facial mesh with anincomplete boundary. FIG. 8B shows the mirrored mesh corresponding tothe mesh of FIG. 8A. FIG. 8C shows the alignment of the meshes of FIGS.8A and 8B by using the ICP algorithm directly, which is not expected.FIG. 8D shows the region between the boundary and the dashed curve onthe mirrored mesh, which region is called the “skirt.” FIG. 8Eillustrates the alignment of the non-skirt region on S_(m) and theoriginal mesh S, which represents the expected correct alignment. FIGS.8F-8H illustrate an example where part of the forehead is missing due tothe occlusion of hair. The vertices colored red in FIG. 8F are theSkirtVertices skirt(s_(m)). FIG. 8G demonstrates the computed symmetryplane without the MarkSkirt process, and FIG. 8H shows the result withthe MarkSkirt process.

FIG. 9 shows mean curvature plots (FIG. 9A) of the four facial surfacesof FIG. 7 and the corresponding colored symmetry profile (FIG. 9B) ofeach facial surface.

FIG. 10 shows the curvature plots and the three essential points of thesymmetry profile of the face surface of FIG. 10A. FIG. 10B shows thecurvature distribution with respect to the arc length. FIG. 10C isgenerated by attaching a line segment along the normal direction at eachpoint of the symmetry profiles. FIG. 10D shows the three essentialpoints P_(NT), P_(NB) and P_(NL) extracted for the face surface.

FIG. 11 shows curvature plots and the three essential points of thesymmetry profile for another example of a face surface, shown in FIG.11A.

FIG. 12 illustrates the Face Intrinsic Coordinate System (FICS), inwhich the y-axis and z-axis lie in the symmetry plane, and the x-axisperpendicular to the symmetry plane.

FIG. 13 shows the SFC representation of an example facial surface,wherein FIG. 13A shows the front view of the facial surface with theprofiles, FIG. 13B shows the isometric view of the profiles and FIG. 13Cshows the isometric view of the SFC representation registered with asecond SFC representation.

FIG. 14 illustrates distance measurements between profiles of two facesthat are registered for comparison according to the present invention,wherein the two profiles differ in length.

FIG. 15 shows scans of multiple expressions for an individual. FIG. 15Ashows the textured 3D meshes directly output by the scanner, and FIG. 3Bshows the corresponding clean faces without texture.

FIG. 16 shows similarity metric distribution in authentication testsconducted according to our invention.

FIG. 17 shows ROC curves of different metrics in authentication testsconducting according to our invention.

FIG. 18 shows two scans of part of the tested individuals used in facerecognition experiment according to our invention. FIG. 18A shows thescans in the database, and FIG. 18B shows the corresponding scans to berecognized.

FIG. 19 shows performance results in the face recognition tests.

DESCRIPTION

FIG. 1 illustrates in schematic block diagram form an exemplary computernetwork system 100 for storing, archiving, query and retrieval ofinformation relating to 3D facial features according to the presentinvention. The computer network system 100 includes a server computersystem 102 and a client computer system 104, which are connected by dataconnections 105, 107 to a computer network 106, e.g., an intranet, theInternet and/or the World Wide Web, so that the client computer system104 and the server computer system 102 can communicate. As will bereadily apparent to persons skilled in the art, the client computersystem 104 is intended to be representative of a plurality of clientcomputer systems 104, each of which may communicate with the servercomputer system 102 via the network 106, whether sequentially orsimultaneously. The computer network system 100 advantageously makes useof standard Internet protocols including TCP/IP and HTTP. TCP/IP is acommon transport layer protocol used by a worldwide network ofcomputers. Although the client 104 and the server computer 102 arecoupled together via the Internet, the invention may also be implementedover other public or private networks or may be employed through adirect connection and any such communication implementation iscontemplated as falling within the scope of the present invention.

The server computer system 102, which has a conventional architecture,includes: a central processing unit (CPU), which may be implemented witha conventional microprocessor; means for temporary storage ofinformation, which may be implemented with random access memory (RAM);and means for permanent storage of information, which may be implementedwith read only memory (ROM); and means for mass storage of information,which may be implemented by hard drive or any other suitable means ofmass storage known in the art.

It will be obvious to someone of ordinary skill in the art that theinvention can be used in a variety of other system architectures. Asdescribed herein, the exemplary system architecture is for descriptivepurposes only. Although the description may refer to terms commonly usedin describing particular computer system architectures the descriptionand concepts equally apply to other computer network systems, includingsystems having architectures dissimilar to that shown in FIG. 1.

Still referring to FIG. 1, the server computer system 102 includes a Webserver 103, and a database server 105. The Web server 103 managesnetwork resources and handles all application operations between thebrowser-based clients 104 and the server side applications. The databaseserver 105 includes a database management system (DBMS), a collection ofprograms that enables the storing, modification and extraction ofinformation from databases 114. The Web server 103 facilitatescommunication and data exchange between the client 104 and database 114.

One or more data acquisition devices 130 can be used to generate raw 3Ddata from a face and to input the raw 3D data to the server 102. Someexamples of suitable data acquisition devices 130 include laser scannersfor acquiring 3D surface data. It will be understood by those of skillin the art, however, that the data acquisition devices 130 can includeany device that generates digitized 3D data from a face. For example,the data acquisition device is not limited the facial surface obtainedby a specific type of scanner.

A program kernal 116 executes on the server computer system 102 andimplements the logical operations of the system, as described below. Thekernal 116 includes a geometric modeling module 118, which can derive 3Dand 2D modeled data from the raw 3D data. A feature extraction, analysisand indexing module 120 can operate to provide cataloging, descriptionand interactive access to the modeled data, as well as to stored textualand descriptive data about the object. A data compression module 121 cancompress certain data for enhanced storage and transmission. A regioneditor program 132 also executes on the server computer system 102. Theregion editor program 132 functions to break the mesh down into“meaningful” connected subsets of vertices called “regions.”

The database 114 can comprise a number of different databases forstoring various data element. These data elements include: 3D acquireddata 122, such as that generated by the data acquisition device 130;modeled data 124, which the geometric modeling module 118 generates fromthe 3D acquired data; derived data 125, which the extraction, analysisand indexing module derives from the modeled data 124, and textual andnumeric data 126. The database 114 also stores metadata 128 fordescribing the data and how the data is formatted. The schema discussedherein further describes the metadata 128.

The 3D acquired data 122 can be 3D surface data, such as that generatedby optically scanning the surface of a face. The modeled data 124 isdata that has been structured or modeled from the acquired data 122using suitable models, such as triangular meshes or surfacerepresentation models. The derived data 125 includes data derived fromthe modeled data, such as object features or mathematical descriptivedate extracted from the modeled data. The text and numeric data 126 caninclude, for example, area curvature distribution and the like.

The data elements are organized, cataloged and described according to aschema to facilitate efficient archiving, query of and interaction withthe data. In one preferred embodiment, the schema is written in XML(extensible Markup Language). FIG. 2 depicts an example of an XML schemafor 3D facial features. The 3D information data structures at thetop-level house original binary data files, modeled files, and derivedfeatures.

Still referring to FIG. 1, the client 104, which also has a conventionalarchitecture, includes: a CPU, which may be implemented with aconventional microprocessor; means for temporary storage of information,which may be implemented with RAM; and means for permanent storage ofinformation, which may be implemented with ROM; and means for massstorage of information, which may be implemented by hard drive or anyother suitable means of mass storage known in the art; one or more inputdevices for inputting information into the computer, which may beimplemented using a conventional keyboard and mouse, and an outputdevice for display graphical output to the user, such as a conventionalmonitor. The client 104 includes conventional hardware and software forcommunicating over the network 106. In a preferred embodiment, thisincludes a Web browser software application 110, which executes on theclient 104 to access information available on the Internet. The systemand method of the present disclosure advantageously utilizes the fullfunctionality of such browser programs, as are known to persons skilledin the operation and use thereof.

3D Face Data Acquisition. According to one preferred method of theinvention, the 3D facial surface we are dealing with is a triangularmesh, which is the most popular representation of 3D geometry. Quite afew types of 3D scanners can output triangular face meshes directly orby means of some surface reconstruction software. 3D scanners operatemuch like 2D cameras, except that the image results in a 3-dimensionalpoint cloud that models the face. The scanning process is similar to asnapshot portrait: the 3D image is acquired instantly and is extremelyunobtrusive. According to one preferred method of the present invention,a point cloud is captured using a commercially available camera, thensoftware produces a meshed surface that accurately models the face as3-D triangle mesh geometry. FIG. 3 shows an example of a 3D camera scanof a face (FIG. 3A) and its corresponding triangle mesh geometry (FIG.3B). The conversion of point clouds into triangle meshes is well known.This process is described in more detail in H. Hoppe, T. DeRose, T.Duchamp, M. Halstead, H Jin, J. McDonald, J. Schweitzer, and Stuetzle,W., Piecewise Smooth Surface Reconstruction, SIGGRAPH 94, ACM Press, pp.295-302 (1994) and A. Razdan, B. Steinberg, G. Farin, From DigitizedData to NURB Surface Meshes, Proceedings of the International Conferenceof Rapid Prototyping and Manufacturing, pp 749-754, Beijing, China(1998), which are incorporated herein in their entirety by thisreference. Preferably, the 3D surface data points are less than 300microns (0.3 mm) apart, providing the capability of distinguishingbetween small, subtle differences regardless of lighting or orientationduring the original scan. The resulting data is representative of thesurface being scanned (a face in this case).

Intelligent Archiving and Data Management. According to a preferredmethod of the invention, the raw 3D face data is archived withadditional semantic information. As previously described, segmentationof 3D features provides the basis for creating feature based indices forarchival and query. For example, the nose serves as a facial feature,and its surface area and curvature distribution make up some of themetadata information.

The database structures preferably take advantage of the latestobject-oriented advances and the established relational technology.Databases such as Oracle provide the well-known advantages of relationaldatabases with extended support for objects, such as storing thebehavior of the object as methods. Since the 3D facial data areinherently objects, the database design can take advantage of thestructure of the data itself.

The flexible, scalable model for complex 3D facial data preferablyprovides: a hierarchy of categories, metadata and shared descriptivevocabulary to provide authority control over data entry, retrieval anddisplay; support for supplementary information associated with the 3Dfacial data, e.g., reference to identification information such as name,social security/id etc. that are generally present in databases; supportfor reference descriptive geometric measurements and metrics such ascurvature, volume, scale, linear dimensions; a model for archiving andstoring of 3D facial data, addressing issues such as file formats,compression, media, disaster recovery, and migration to alternateformats. Once the 3D facial data has been modeled and features have beenextracted, these elements can be automatically catalogued and describedusing the schema developed for facial data.

According to a preferred embodiment, class models for face data type areimplemented as XML document type declarations (DTDs) that define theschema for a particular XML file type. The DTDs are designed to beconsistent with emerging standards (including Dublin Core, W3C), andfacial data standards. FIG. 2 shows a sample XML schema that can be usedin connection with the invention.

According to the present invention, the bilateral symmetry of a humanface is an important global feature for understanding the face. Similarto Beumier, et al., our approach is based on facial bilateral symmetryextraction and thereby the symmetry profile as well, but we use adifferent method based on mirror reflection and model registration. C.Beumier, M. Acheroy, Automatic 3D face authentication. Image and VisionComputing, 18:315-321 (2000). Referring to FIG. 6, a preferred methodaccording to our invention by extracting the bilateral symmetry planethe facial mesh (step 200). A symmetry profile is computed (step 202).The symmetry profile represents the intersection between the symmetryplane and the facial mesh. In addition to the symmetry profile, atransversal forehead profile for the facial mesh is then computed (step202) and a transversal cheek profile for the facial mesh is computed(step 202). The symmetry profile, the forehead profile and the cheekprofile are stored as a compact representation of the surface of theface to be authenticated (step 203). We use a novel method based on boththe mean curvature plot of the facial surface and the curvature plot ofthe symmetry profile curve to accurately extract three points of thenose on the symmetry profile (step 204). The three extracted pointsuniquely determine a Face Intrinsic Coordinate System (FICS). The faceintrinsic coordinate system is used to align the compact representationof the surface of the face to be authenticated with one or more storedcompact representations of faces (step 206). The compact representationof the surface of the face to be authenticated is then compared with theone or more stored compact representations of faces (step 208).

Symmetry Plane Extraction

A 3D object is bilaterally symmetric if there exists some plane, suchthat the object is invariant under reflection through that plane.

Bilateral symmetry computation of shapes and images is a basic problemin the field of mathematics, computer vision, image processing and thelike. Most symmetry extraction algorithms are designed for 2Dapplications. Examples of such algorithms are described by D. Reisfeld,H. Wolfson, Y. Yeshurun, Detection of interest points using symmetry,Proc. of the ICCV'1990. pp. 62-65, and Y. Liu, K. L. Schmidt, J. F.Cohn, et al., Facial asymmetry quantification for expression invarianthuman identification, Computer Vision and Image Understanding. 91(1,2):138-159 (2003). Only some of these algorithms, however, can be extendedto 3D. H. Zabrodsky, S. Peleg, D. Avnir, Symmetry as a continuousfeature, EEE transactions on Pattern Analysis and Machine Intelligence,17:1154-1166 (1995). One category of methods computes the 3D bilateralsymmetry plane by using the principal axes of inertia, i.e. theeigenvectors of the covariance matrix of the point distribution. Forexample, O'Mara et al used this idea to measure bilateral symmetry inthree dimensional magnetic resonance images of human heads. D. O'Mara,R. Owens, Measuring bilateral symmetry in three dimensional magneticresonance images, TENCON Digital Signal Processing Applications,Piscataway, U.S.A., pp 151-156 (1996). Tuzikov et al also used theprincipal axes of inertia to define the brain symmetry plane in MRimages. A. V. Tuzikov, O. Colliot, I. Bloch, Brain symmetry planecomputation in MR images using inertia axes and optimization, ICPR12002Quebec, Canada. But in their approach, an optimization process (thedownhill simplex method) was further used to find a more accuratesymmetry plane.

Another category of methods treats the bilateral symmetry extraction asa registration problem between the original and the reflected images.Benz, et al used this method to compute the symmetry plane of a humanface. M. Benz, X. Laboureux, T. Maier, et al., The Symmetry of Faces,Proceedings of VMV'2002, Erlangen, Germany. They first mirror the wholedataset of a face, which is represented in a 3D triangular mesh, at anarbitrary plane. Then the ICP (Iterative Closest Point) algorithm wasused to align the original and the mirrored facial surfaces. For a moredetailed discussion of this process, see P. J. Besl, N. D. McKay, Amethod for registration of 3-D shapes, IEEE Trans. on Pattern Analysisand Machine Intelligence, 14(2):239-256 (1992). Based on the registeredtwo datasets, the symmetry plane of the face can be easily computed.Their study is also for a clinical application—to support the surgeonintraoperatively during the repair of a displacement of the eye globe ofa patient. Because the ICP algorithm is sensitive to the initialrelative position of the two shapes to be registered, however, they haveto manually coarsely register the two data sets before running the ICPalgorithm.

We use the basic idea of mirror transformation followed by aregistration procedure to extract the symmetry plane of a human face. Tomake the entire process totally automatic and pose-invariant, we firstdetermine the most appropriate position of the mirror plane by means ofthe principal component analysis (PCA), so that the original mesh andthe mirrored mesh are approximately aligned. The ICP algorithm is thenused to get a refined registration.

Mirror Plane Determination

Let S(P, K) denote the triangular facial mesh surface, where P is a setof N point positions p^(i) (xi, yi, zi)εR³, 1≦i≦N, and K is an abstractsimplicial complex which contains the adjacency information of vertices:v={i}εK, edges: e={i, j}εK and triangular facets: f={i,j,k}εK.

Suppose O_(s) is the centroid of the facial mesh S(P, K). We firstconstruct the covariance matrix of the vertex distribution as follows:$\begin{matrix}{C = {\sum\limits_{i = 0}^{N}\quad{\left( {p^{i} - O_{s}} \right)\left( {p^{i} - O_{s}} \right)^{T}}}} & (1)\end{matrix}$

By means of principal component analysis, we can get three eigenvaluesλ₁≧λ₂≧λ₃ and the three corresponding eigenvectors v₁, v₂ and v₃ of C.The facial surface we are dealing with is the frontal part of the headand the dimension in up/down direction is typically longer than that inleft/right direction. Thus, the covariance matrix C is expected to havethree different eigenvalues. The eigenvector v₃ represents the normaldirection of the least square fitted plane of the facial surface. Theeigenvector v₁ corresponds to the up/down direction of the facialsurface. The complexity of the PCA algorithm is O(N).

Taking O_(s) as the origin, v₁ as the y-axis, v₃ as the z-axis, wedefine a new right-hand coordinate system. This coordinate system iscalled the Pose Coordinate System (PCS), for it represents the head poseand depends only on the vertex distribution of the facial surface. Theoriginal facial surface S(P,K) is first transformed to the posecoordinate system. We then use the yz coordinate plane of the PCS as themirror plane to get a mirrored facial surface S_(m). This procedure canbe described as $\begin{matrix}{\left. p_{m}^{i}\leftarrow{R \cdot A \cdot \left( {p^{i} - O_{s}} \right)} \right.\left. p^{i}\leftarrow{A \cdot \left( {p^{i} - O_{s}} \right)} \right.{{R = \begin{bmatrix}{- 1} & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{bmatrix}},\quad{A = \begin{bmatrix}v_{2}^{x} & v_{1}^{x} & v_{3}^{x} \\v_{2}^{y} & v_{1}^{y} & v_{3}^{y} \\v_{2}^{z} & v_{1}^{z} & v_{3}^{z}\end{bmatrix}}}} & (2)\end{matrix}$where p_(m) ^(i) is the mirrored point of p^(i); A is the rotationmatrix from the original coordinate system to the PCS; R is thereflection matrix with respect to the mirror plane; v in matrix Arepresents the components of the eigenvectors v₁, v₂ and v₃ of C.

In case the face dimension in the up/down direction is shorter than orapproximately equal to that in the left/right direction (of all the 213facial meshes we tested, no such case arose), the xz instead of yzcoordinate plane of the pose coordinate system may be the best mirrorplane. This can be solved by trying both xz and yz coordinate plane asthe mirror plane, and selecting the one that leads to smaller residualmean distance in the model registration procedure described below.

FIG. 7 shows the results of the intersection between the estimatedmirror plane and the facial mesh surface on four individuals. FIG. 7Billustrates the intersection between the mirror planes (by the PCAalgorithm) and the facial meshes, and FIG. 7B illustrates theintersection between the automatically extracted symmetry planes and thefacial meshes (by aligning the mirrored and the original facial meshes.It is known that if an object is bilaterally symmetric, then thesymmetry plane passes through its centroid and is orthogonal to someprinciple axis. FIG. 7 demonstrates that the mirror plane computed byusing PCA is close to the symmetry plane but not accurate in some cases.This can be explained by three reasons. First, the face itself is notperfectly symmetric. Second, the centroid O_(s) is obtained by averagingall the vertices in the facial mesh S, but the distribution of thevertices of the meshes is not perfectly even. Third, and most important,the boundary of the facial mesh is irregular and might be quiteasymmetric.

Model Registration

Since the mirror plane does not coincide with the real symmetry plane inmost cases, there exists some misalignment between the mirrored meshS_(m) and the original mesh S. To get an optimized registration, we usethe basic idea of the Iterative Closest Point (ICP) algorithm. The ICPalgorithm is described in more detail by P. J. Besl, N. D. McKay, Amethod for registration of 3-D shapes, IEEE Transactions on PatternAnalysis and Machine Intelligence, 14(2):239-256 (1992), which isincorporated herein in its entirety by this reference.

Given two 3D point sets C, and C₂, the main steps of the ICP algorithmare:

-   -   (1) For every point in C₂, compute its closest point in C₁.    -   (2) The mean square distance metric of the pairs of closest        points are minimized with respect to a rigid body transform.    -   (3) Apply the transform to C₂.    -   (4) Terminate the iteration if the change in mean square        distance falls below a preset threshold, otherwise, return to        Step (1).

In our case, the two data sets are vertices of the facial meshes S andS_(m). Since the boundary of the facial mesh could be irregular (see,e.g. FIG. 8A), S and S_(m) may be misaligned (see FIG. 8C), which leadsto an incorrect symmetry plane. To solve this problem, we employ aprocedure called MarkSkirt.

First, we introduce the neighborhood relation nbhd:nbhd{i}:={i}∪{j|∃{i, j}εK}nbhd{i ₁ ,i ₂ , . . ._(i)k}:=°_(μ−1, . . . ,k) nbhd{i _(μ)}  (3)

We also use the following recursive definition:nbhd ¹ {i}:=nbhd{i}nbhd ^(n) {i}:=nbhd{nbhd ^(n−1) {i}}, n≧2  (4)

-   -   nbhd_(n){i} is also called the n-ring neighbors of vertex {i}.        The MarkSkirt procedure finds and marks the n-ring neighbors of        all the vertices on the boundary of the mirrored mesh S_(m) as        Skirt. This is defined as        Skirt(S _(m)):={j|jε∪ _({i}ε∂S) _(m) nbhd ^(n) {i}}  (5)        where ∂s_(m) represents the boundary of S_(m).

After the MarkSkirt procedure, we obtain:P _(m) ′=P _(m)\Skirt(S _(m))  (6)where P_(m) denotes the vertex set of S_(m). Only the vertices in P_(m)′are sampled to match with the original mesh S. This is to make sure thatthe region to be matched with the original mesh S is the subset of S(see FIG. 2 E), for the ICP algorithm has good performance when thegeometry of one data set is the subset of the geometry to be matched to.In our experiments, 10-ring neighbors of the boundary are marked asSkirt. This procedure is crucial to get a correct alignment. Ourexperiments show about 15 percent of unexpected alignment without theMarkSkirt process. With the MarkSkirt process, however, all the 213 facedata sets get visually pleasing symmetry planes (some of the testedfaces have very special and asymmetric expressions). The appearanceitself of some individuals is slightly asymmetric, but the intrinsicasymmetry is reproductive in different scans, and hence will not affectthe decision making in authentication and recognition tasks.

FIG. 8 illustrates the MarkSkirt procedure. FIG. 8A shows a 3D facialmesh with an incomplete boundary. FIG. 8B shows the mirrored meshcorresponding to the mesh of FIG. 8A. FIG. 8C shows the alignment of themeshes of FIGS. 8A and 8B by using the ICP algorithm directly, which isnot expected. FIG. 8D shows the region between the boundary and thedashed curve on the mirrored mesh, which region is called the “skirt.”FIG. 8E illustrates the alignment of the non-skirt region on S_(m) andthe original mesh S, which represents the expected correct alignment.FIGS. 8F-8H illustrate an example where part of the forehead is missingdue to the occlusion of hair. The vertices colored red in FIG. 8F arethe SkirtVertices skirt(s_(m)). FIG. 8G demonstrates the computedsymmetry plane without the MarkSkirt process; and FIG. 8H shows theresult with the MarkSkirt process.

Referring again to the steps of the ICP algorithm, the complexity inSteps (2) and (3) are O(N²). But in Step (1), the complexity forcomputing the entire set of the closest points is O(N²), and thecomputation needs to be performed in every iteration. The ICP algorithmis normally time-consuming due to the cost of finding the closestpoints. According to a preferred method of our invention, two proceduresare used to speed up the ICP algorithm. First, partial instead of entirevertices of P_(m)′ are randomly sampled to match with the original meshS. In our experiments, only 5% of the vertices in P_(m)′ are used. Thisis reasonable also because of the fact that the ICP algorithm has goodperformance when one data set is the subset of the other one. We furtherspeed up the closest points finding process by using a spacepartitioning scheme. In the mirror plane determination process describedabove, we transformed the facial mesh S to the Pose Coordinate System.Now, we construct in PCS an axis-aligned bounding box that encloses Sand S_(m). The bounding box is then partitioned into m×n×l cells. Andthe searching for the closest point of vertex {i} E S_(m) can be limitedto the closest cells surrounding {i}. In our experiments, the number ofcells m in the x-direction (corresponding to the approximate left/rightdirection) is set to 10. Suppose the edge lengths of the bounding box inx, y and z directions are l_(x), l_(y), l_(z) respectively, thenn=10*l_(y)/l_(x), and l₌10*l_(z)/l_(x).

Our experiments show that the above mentioned schemes for acceleratingthe algorithm are effective. The average time spent on the process ofICP is 1.6 seconds (See Section 5.4 for more information on computationtime). The good initial relative position of S and S_(m), due to theadequate mirror plane computed by the PCA, not only gives more chancesto achieve the global optimized registration, but also makescontribution to the rapid convergence of the ICP algorithm. Table 1illustrates the mean distance between the pairs of closest points of theoriginal mesh and the mirrored one for four persons in FIG. 1. From thevalues in the table we can see that the ICP procedure improves thealignment at various extents for different persons. We also find that ifthe boundary of the facial mesh is good enough (e.g. that of the personshown in FIG. 7D), the PCA algorithm itself can generate a quite goodmirror plane, which is very close to the bilateral symmetry plane of theface. TABLE 1 Mean distance between the original mesh and the mirroredone, before and after the ICP algorithm. Mean Dis. (mm) Person 1 Person2 Person 3 Person 4 Before ICP 3.088 2.329 1.162 1.162 After ICP 1.3161.103 1.030 0.958Symmetry Profile Extraction

After the original facial surface S and the mirrored surface S_(m) havebeen registered, the symmetry plane can be fitted easily. Define$\begin{matrix}{{p_{c}^{i} = {\frac{1}{2}\left( {{R_{ICP} \cdot \left( {p_{m}^{i} - t_{ICP}} \right)} + p^{i}} \right)}},\quad{i = 1},2,\ldots\quad,N} & (7)\end{matrix}$where R_(ICP) and t_(ICP) are the rotation matrix and the translatevector output by the ICP algorithm respectively. The midpoints p_(c)^(i), i=1,2, . . . ,N, are used to fit a least square plane, which isthe final symmetry plane, denoted as PLS. The symmetry profile of a faceis extracted by intersecting PL^(S) with the facial surface S. Thecomplexity of this process is O(√{square root over (N)}).

FIG. 7B shows the final symmetry planes of each of the four facialsurfaces of FIG. 7A.

Symmetry Profile Analysis

The symmetry profile extraction takes us much closer to recognizing aface. The symmetry profile passes through the center of the forehead,the ridge of the nose, the philtrum, the mouth and the chin. However,the nose seems to be the most robust geometrical feature on the symmetryprofile and even on the entire face. It is least changed under differentexpressions and contains clearly distinguishable points. We attempted tounderstand the symmetry profile by only analyzing the curvature of theprofile curve. But different individuals show quite different curvatureplots of the symmetry profile curves, including the number of inflectionpoints. Some profiles go from the forehead to the chin; the others gofrom the chin to the forehead. So, the sign of the curvature at specificpoints, e.g. the nose tip, might be different among symmetry profiles.The point corresponding to the largest curvature might be the nose tip,the upper or lower mouth lip, and the conjunction point between the noseand the philtrum, and so on. Because of this, we use clues from both thefacial surface curvature analysis and the symmetry profile curvatureanalysis.

Mean Curvature Plot of the Facial Surface

We first roughly isolate the nose from its surrounding area by means offacial surface curvature analysis. According to a preferred method, weuse the mean curvature plot to isolate the nose. It will be understoodto those of skill in the art, however, that other analysis methods canbe used to achieve this. Some other examples of such methods include theprinciple curvature plot, the Gaussian curvature plot and the HK segmentmethod. The experiments show, however, that the mean curvature plot isthe most effective method to isolate the nose.

We calculate the mean curvature on the facial mesh S as follows. Takingthe centroid of each triangle {i,j,k} in S as the origin, we establish aquadratic surfaceQ(u,v)=(u,v,h(u,v))  (8)Whereh(u,v)=au ² +buv+cv ²  (9)axis h points to the normal direction of that triangle, u and v are inthe triangle plane and orthogonal to each other. By substituting thevertices of {i}, {j},{k} and their 1-ring neighbor vertices to thequadratic surface Q(u,v), the parameters a, b and c can be determined byusing least square fitting. The curvature at Q(0,0) is taken as thecurvature at the centroid of triangle {i,j,k}. According to differentialgeometry, the mean curvature at Q(0,0) isH=a+c  (10)

We classified the curvature values into three scales. Define$\begin{matrix}\left\{ \begin{matrix}{H_{1} = {H_{\min\quad} + {\frac{1}{3}\left( {H_{\max} - H_{\min}} \right)}}} \\{H_{2} = {H_{\min} + {\frac{2}{3}\left( {H_{\max} - H_{\min}} \right)}}}\end{matrix} \right. & (11)\end{matrix}$where H_(max) and H_(min) are the maximum and minimum mean curvaturevalues. To eliminate the effect of noise, the 1% highest and lowest meancurvature values are not taken into account in searching for H_(max) andH_(min).

For triangle T_(i), a color attribute is attached to it by$\begin{matrix}{{{color}\left( T_{i} \right)} = \left\{ \begin{matrix}{{``{blue}"},\quad{{{if}\quad H_{i}} < H_{1}}} \\{{``{red}"},\quad{{{if}\quad H_{1}} \leq H_{i} \leq H_{2}}} \\{{``{green}"},\quad{{{if}\quad H_{1}} > H_{2}}}\end{matrix} \right.} & (12)\end{matrix}$where H_(i) is the mean curvature value at the centroid of T_(i).

FIG. 9A shows the mean curvature plots, in terms of the color attribute,of each of the four faces of FIG. 7A. Colored regions which are isolatedand smaller than a criterion are automatically merged to theirsurrounding regions. In FIG. 9A, the mean curvature of the nose of eachface is blue. Our experiments show that the nose in the mean curvatureplot is almost always blue (except for only one out of the 213 testedfacial meshes), despite the large appearance variation amongindividuals. The mean curvature plot is used, as described below, toextract some essential points on the symmetry profile.

Curvature Analysis of the Symmetry Profile

For analyzing the symmetry profile, we also need to calculate thecurvature of the symmetry profile curve. Since the symmetry profile isgenerated by intersecting the symmetry plane with the facial mesh S, itis represented by a planar polyline. The curvature at each vertex x_(i)on the polyline is calculated by locally fitting a least square parabolaq(t)=(t,f(t))  (13)wheref(t)=a _(c) t ² +b _(c) t+c _(c)  (14)

Axis f and t are the binormal and the tangent direction of the symmetryprofile curve at x_(i) respectively. By substituting the localcoordinates of vertices x_(j), j=i−2, . . . ,i+2 to the parabola q(t),the parameters a_(c), b_(c) and c_(c) in Eq. (14) can be determined byusing the least square fitting. The curvature at x_(j), i.e. q(0), isκ=2a _(c)/(1+√{square root over (b _(c) ³ )})  (15)

FIGS. 10 and 11 show curvature plots of the symmetry profiles of theface surfaces of two individuals. FIGS. 10A and 11A show the facesurfaces. FIGS. 10B and 11B show the curvature distribution with respectto the arc length for the face surfaces of FIGS. 10A and 11A,respectively. FIGS. 10C and 11C show the curvature distribution alongthe symmetry profile of the faces of FIGS. 10A and 11A, respectively.Each of these curvature distributions was generated by projecting a linesegment along the normal direction at each point of the symmetryprofiles of the face surface. The length of the segment is proportionalto the signed curvature at that point. From FIGS. 10C and 11C, we canalso see that the signs of the curvature at specific points, e.g. at thenose tips, are different in the two examples. This is because oneprofile is from the forehead to the chin, and the other is from the chinto the forehead.

Essential Points Recognition

Using the mean curvature of the facial surface and the curvature of thesymmetry profile, we can recognize three essential points on thesymmetry profile, namely the nose tip (denoted as P_(NT)), the nosebridge (denoted as P_(NB)), and the lowest point of the nose (denoted asP_(NL)). FIGS. 10D and 11D show the three essential points P_(NT),P_(NB) and P_(NL) for each of the face surfaces of FIGS. 10A and 11A,respectively.

The points P_(NT), P_(NB) and P_(NL) are automatically extracted asfollows. First, the color of the facial surface (see FIG. 9A) isinherited by the symmetry profile, i.e., if a line segment in thesymmetry profile is the intersection between the symmetry plane and atriangle in the facial mesh S, the color of that line segment is thesame as the color of that triangle. In this way, we get a coloredsymmetry profile, as shown in FIG. 9B. This is exactly the way we usethe mean curvature plot. We only make use of the mean curvatureinformation of the facial surface along the symmetry profile.

We then use some prior knowledge to extract the three essential pointsP_(NT), P_(NB) and P_(NL) on the symmetry profile. Suppose the symmetryprofile is represented by a normal arc-length parameterized curve C(s).The first essential point to be extracted is the nose tip P_(NT). In thecolored symmetry profile, the nose is a relatively long blue segment.The only other part that might be confused with the nose is the chin,which can also be a relatively long blue segment. However, the nosesegment is always located closer to the center point C(0.5) of thesymmetry profile than the chin is. That distinguishes the nose segmentfrom the other geometric features in the symmetry profile. Suppose theblue nose segment is denoted as [s₀ ^(n), s₁ ^(n)]. We take the pointwith the largest absolute curvature value κ in that segment as the nosetip P_(NT), i.e. $\begin{matrix}{s_{NT} = {\arg\quad\underset{s \in {({s_{0}^{n},s_{1}^{n}})}}{\max\left( {{k(s)}} \right)}}} & (16)\end{matrix}$If the curvature value at P_(NT) is negative, the curvature value ateach vertex on the profile is multiplied by −1.

The symmetry profile can be regarded as two parts separated at P_(NT).The part from P_(NT) to the end of the forehead is called the upperpart; the part from P_(NT) to the end of the chin is called the lowerpart. If |s_(NT)−s₀ ^(n)|<|s_(NT)−s₁ ^(n)|, the direction from s_(NT) tos₀ ^(n) is towards the chin, otherwise, the direction from s_(NT) to s₀^(n) is towards the forehead. In this way, we distinguish the upper partand the lower part. Taking P_(NT) as the starting point, we search inthe lower part for the first point with local minimal curvature value κ.That point is taken as P_(NL). We then search in the entire upper partfor the point with the smallest curvature value κ. That point is takento be P_(NB). It should be noticed that, taking P_(NT) as the startingpoint, P_(NB) is not necessarily the first point with local minimalcurvature in the upper part, because some people have extra localminimal curvature values on their noses, as shown in FIGS. 10 and 11.

From the above description we can see that the requirement for coloringthe facial mesh and the symmetry profile is very loose. Differently fromthe other face segmentation methods based on curvature analysis, themean curvature plot in our approach is only used to roughly estimate asegment on the symmetry profile within which the nose tip is located.The mean curvature plot of the facial surface is robust enough for thisloose requirement. On the other hand, the colored symmetry profile doesprovide a basic and important clue for locating the nose tip anddetermining the up/down direction of the profile.

Similarity Comparison

As shown in FIG. 12, the three essential points P_(NT), P_(NB), P_(NL)uniquely determine a coordinate system. With P_(NT) is the origin,vector V_(TB) represents the direction of the y-axis, whereV_(TB)=(P_(NT)−P_(NB))/∥P_(NT)−P_(NB)∥. The cross product of V_(TB) andV_(TL) coincide with the direction of the x-axis, whereV_(TL)=(P_(NT)−P_(NL))/∥P_(NT)−P_(NL)∥. The z-axis is determined by theright-hand rule. This coordinate system we call the Face IntrinsicCoordinate System (FICS).

There are multiple choices for computing the similarity between faces. Agood choice should satisfy the following criteria:

It should be discriminative enough.

It should be computationally efficient.

It should lead to a concise representation of the face so as to reducethe space size of the face database.

According to a preferred method, we use the mean distance between thealigned representations of the faces that we define as the SFC facerepresentations. The SFC representation includes three profiles on thefacial surface, namely the symmetry profile, the forehead profile andthe cheek profile.

FIG. 13 shows the SFC representation of an example facial surface, withFIG. 13A showing the front view of the facial surface with the profilesand FIG. 13B showing the isometric view of the profiles. The transversalforehead profile is the intersection between the facial surface and theplane PL(P_(f), n), where P_(f) is the point on the symmetry profile and3 cm away from the nose bridge point P_(NB) in terms of the arc length,and n=P_(NT)−P_(NB) is the normal vector of the plane. The transversalcheek profile is the intersection between the facial surface and theplane PL(P_(c), n), where P_(c) is the point on the symmetry profile and2 cm away from the nose tip point P_(NT) in terms of the arc length. Thereason for selecting the 3 cm/2 cm points is to avoid the eye regionswhich are not rigid enough. After the SFC representation is obtained, itis transformed to the FICS. Finally the SFC representation, rather thanthe space-consuming facial mesh, is stored in the database.

Since the SFC representations are under their FICSs, they are roughlyaligned. To compute the similarity between two faces, we furtheroptimize the symmetry profile alignment by using the ICP algorithmagain. Only the points between P_(NL) and P_(NB) on the two symmetryprofiles respectively are used to optimize the alignment. The rotationand translation transforms that are output by the ICP algorithm areapplied to the second symmetry profile. The computation expense in thisprocedure is almost negligible, for the number of points between P_(NL)and P_(NB) on the symmetry profile is typically less than 100, and theirinitial relative locations are very close. For the two transversalforehead profiles, we apply a translation to make P_(f1) and P_(f2) aswell as P_(c1) and P_(c2) coincide respectively. FIG. 13C shows anisometric view of two registered SFC representations.

After the SFC representations of two faces are finally registered, thesimilarity metric between them can be calculated. Let L1 and L2 be twoprofiles. Because the distance between two polylines is directional,i.e. the mean distance from L1 to L2 and that from L2 to L1 might bedifferent, we define $\begin{matrix}{E_{1} = {\frac{1}{N_{L\quad 1}}{\sum\limits_{p_{1} \in \quad{L\quad 1}}\quad{{Min}_{p_{2} \in \quad{L\quad 2}}{d\left( {p_{1},p_{2}} \right)}}}}} & (17) \\{E_{2} = {\frac{1}{N_{L\quad 2}}{\sum\limits_{p_{2} \in \quad{L\quad 2}}\quad{{Min}_{p_{1} \in \quad{L\quad 1}}{d\left( {p_{2},p_{1}} \right)}}}}} & (18)\end{matrix}$where N denotes the number of sampled points on the profile, d is theEuclidean distance. The similarity (accurately dissimilarity) metricbetween the two profiles L1 and L2 is defined as: $\begin{matrix}{E = {\frac{1}{2}\left( {E_{1} + E_{2}} \right)}} & (19)\end{matrix}$

FIG. 14 illustrates distance measurements between profiles of two facesthat are registered for comparison according to the present invention.It should be noted that the corresponding profiles of two compared facesmay be different in length, as shown in FIG. 14. In such a case, thelonger parts of the profiles (see the gray segment in FIG. 14) areautomatically truncated for similarity computation.

According to Eq. (17)˜Eq. (19), we compute three similarity metricsE_(S), E_(F) and E_(C), where E_(S) is the mean distances between thesymmetry profiles, E_(F) is the mean distance between the foreheadprofiles and the E_(C) is the mean distance between the cheek profiles.Below we discuss experimental results on face authentication using thethree metrics and their weighted combination defined as:E _(W) =w _(S) E _(S) +w _(F) E _(F) +w _(C) E _(c)  (20)where w represents the weight of a metric.

It is straightforward to offset the profile planes to get more verticaland/or transversal profiles. Our experiments showed, however, thatrecognition accuracy would decrease instead of increase when too manyprofiles were used, especially in case of extreme expressions.

EXPERIMENTAL RESULTS

The following test examples and results help to further explain theinvention. It will be understood, however, that the examples areillustrative of the invention and that the invention is not limited onlyto these examples.

Our method was tested on 213 face surfaces, which came from 164individuals and covered a wide ethnic variety and different expressions.The Equal Error Rate (EER) of face authentication on the tested faceswas 3.8% and the rank one recognition rate was 90%.

Data Acquisition

We conducted experiments with the system and method of our inventionusing a 2-pod Qlonerator scanning system from 3Q Technologies, Ltd. OfAtlanta, Ga. (http://www.3q.com/) to scan faces. The Qlonerator systemoutputs textured 3D triangular mesh surfaces in natural lightenvironment. We discarded the texture information and kept thetriangular mesh in our algorithm. This is partly because the textureinformation is not reliable in some cases. Another consideration is thatusing only the triangular mesh makes the algorithm suitable for dealingwith datasets obtained by other types of scanner.

We scanned 164 individuals in our experiments. Twenty individuals hadmore than two scans. One of them had 20 scans with differentexpressions. FIG. 15 shows scans of multiple expressions for the sameperson for an individual. We investigated 213 total facial meshes, whichcovered a wide ethnic and age variety, different facial hair conditionsand various expressions. No one in our experiments wore glasses.

Preprocessing of original scans involved: (1) semiautomatic trimming ofthe original mesh to remove the extraneous regions, leaving only theface; (2) automatically relaxing the triangular mesh to make it manifoldand smoother; and (3) automatically filling holes and removing spikes inthe surface. After preprocessing, we obtained clean facial meshes. FIG.15A shows examples of textured 3D meshes directly output by the scanner,and FIG. 3B shows the corresponding clean faces without texture.

Authentication

Face authentication involves performing verification based on aone-to-one search to validate the identity claim of the individual(i.e., access control for a building, room, or for making a transactionat an ATM terminal). In the applications of face authentication, adissimilarity threshold should be determined for accepting or rejectinga claim.

In our facial authentication experiments, we compared 300 pairs of facesfrom different persons, and 200 pairs of faces from the same persons. Toobtain the different person comparisons, three faces were randomlyselected from the 164 scans in the database, and each was compared withall the other scans. 300 out of the 489 (3×163) different personcomparisons were randomly chosen to conduct the statistical analysis. Toobtain the same person comparisons, each pair of the 20 scans of aperson were compared, which generates 190 comparisons. FIG. 8illustrates some of the expressions and postures of the 20 scans. Tenmore same person comparisons were performed on ten different individualswho had two scans.

Here we provide some statistical terminologies for face authentication.Given pairs of facial meshes of the same person, Error Rejection Rate(ERR) is the percentage that is falsely rejected as different persons.Given pairs of facial meshes of different persons, Error Acceptance Rate(EAR) is the percentage that is falsely accepted as the same person. IfERR=EAR under a threshold of a specific metric, then that percentage iscalled Equal Error Rate, which can be used to evaluate the performanceof the comparison algorithm. C. Beumier, M. Acheroy. Automatic 3D faceauthentication. Image and Vision Computing, 18:315-321 (2000).

We first investigated the discriminating power of each metric E_(S),E_(F) and E_(C). Our experiments show that E_(F), i.e. the metric ofmean distance between forehead profiles, has the smallest average valueand outperforms E_(S) and E_(S) (see FIG. 16 and FIG. 17). This suggeststhat the weight w_(F) in Eq. (20) should be larger than w_(S) and w_(C).It is possible to optimize the three weights in view of statisticalanalysis. But in all our experiments so far, we set w_(S)=w_(C)=1.0,w_(F)=4.0 for the weighted combination metric E_(w). FIG. 16 illustratesthe histogram of number of pairs with respect to the metrics. FIG. 17shows the face authentication ROC curves of different metrics. The EERof the weighted combination metric E_(W) is 3.8%.

Recognition

While the focus of the present method of our invention is faceauthentication, the method can be used in face recognition tasks aswell. In face recognition applications, one-to-many searches areperformed to identify an unknown individual based on comparison to adatabase of known individuals (e.g., law enforcement and surveillance).Unlike the authentication scenario, face recognition does not need acriterion to determine if two scans represent the same person. Insteadface recognition needs to find the “best match” in terms of somesimilarity metric of a given scanned data from the database.

We performed face recognition experiments on 20 individuals. Thedatabase contained the facial meshes of 164 individuals, which includethe 20 tested persons. One different scan of each tested person wascompared to all the 164 scans in the database. FIG. 18 shows the twoscans of part of the tested individuals in the face recognitionexperiment. The first row is the scans in the database. The second rowis the corresponding scans to be recognized. In this experiment, sometested persons also had different expressions or appearances in theirtwo scans, as shown in FIG. 18. The performance of the similaritymetrics E_(S), E_(F), E_(C) and E_(W) in face recognition experiment isillustrated in FIG. 19. Just as in the authentication experiments, theweighted combination (E_(W)) is still the best metric in facerecognition. But in the recognition experiments, the performance of theforehead profile metric E_(F) was worse than the other two profilemetrics E_(S) and E_(C).

Computation Analysis

The average number of triangles contained in one facial mesh in ourexperiments was about 13,000. It took an average time of 3.8 seconds toobtain the SFC representation from a clean facial mesh. The mosttime-consuming stage was the ICP algorithm for aligning the original andthe mirrored mesh, which took an average time of 1.6 seconds. The secondtime-consuming procedure was the mean curvature computation of thefacial mesh, which took 1.4 seconds. After the face database of SFCrepresentations was established, comparison between two faces took anaverage time of 0.7 second. The times reported in this paper are all theexperimental results on a 1 GHz Pentium IV PC with 512M RAM.

The average size of the SFC representation is about 30 KB, which isabout 30 times smaller than the size of the triangular meshrepresentation.

CONCLUSION

The above-described invention possesses numerous advantages as describedherein. We have demonstrated an approach for human face authenticationand recognition from 3D facial meshes. The method is based on theextraction of the bilateral symmetry plane and the recognition of threeessential points of the face geometry, which prove to be robust toasymmetry boundaries and appearances of the scans. We also have provideda compact representation of the 3D face geometry, namely the SFCrepresentation. Face authentication and face recognition are studied bycomparing the mean distance between SFC representations. The weightedcombination E_(W) of the similarity metrics of the symmetry profile, theforehead profile and the cheek profile generates promisingauthentication and recognition rates under various expressions.Authentication and recognition experiments also show that the falseresults are mainly caused by extreme expressions. The time and spaceconsumption in the proposed method is acceptable in classic facecomparison scenarios.

The invention in its broader aspects is not limited to the specificdetails, representative devices, and illustrative examples shown anddescribed. Those skilled in the art will appreciate that numerouschanges and modifications may be made to the preferred embodiments ofthe invention and that such changes and modifications may be madewithout departing from the spirit of the invention. For example, if thesimilarity metric E_(W) between two scans falls into an interval thatcannot be determined for sure (The interval between the zero EAR andzero ERR. See the similarity metric distribution in FIG. 16), additionalcomparison, e.g. between the nose surfaces, might be performed toimprove the authentication and recognition effects. As another example,although we have obtained promising results using only the 3D geometryinformation, the scan texture information may be used as an additionalclue in face comparison tasks.

Therefore, the invention in its broader aspects is not limited to thespecific details, representative devices, and illustrative examplesshown and described. Accordingly, departures may be made from suchdetails without departing from the spirit or scope of the generalinventive concept.

1. A method for automatic authentication and recognition of a humanface, the method comprising: generating a three-dimensional facial meshrepresenting the surface of a human face to be authenticated; extractingfrom the facial mesh a bilateral symmetry plane; computing a symmetryprofile representing the intersection between the symmetry plane and thefacial mesh; computing a transversal forehead profile for the facialmesh; computing a transversal cheek profile for the facial mesh; storingthe symmetry profile, the forehead profile and the cheek profile as acompact representation of the surface of the face to be authenticated;extracting three points on a portion of the symmetry profilerepresenting the nose, the three points uniquely determining a faceintrinsic coordinate system; using the face intrinsic coordinate systemto align the compact representation of the surface of the face to beauthenticated with one or more stored compact representations of humanfaces; and comparing the compact representation of the surface of theface to be authenticated with the one or more stored compactrepresentations of human faces.